Identification and recognition of remote-controlled malware

Remote-controlled malware, organized in so-called botnets, have emerged as one of the most prolific kinds of malicious software. Although numbers vary, in extreme cases such as Conficker, Bredolab and Mariposa, one botnet can span up to several million infected computers. This way, attackers draw substantial revenue by monetizing their bot-infected computers. This thesis encapsulates research on the detection of botnets – a required step towards the mitigation of botnets. First, we design and implement Sandnet, an observation and monitoring infrastructure to study the botnet phenomenon. Using the results of Sandnet, we evaluate detection approaches based on traffic analysis and rogue visual monetization. While traditionally, malware authors designed their botnet command and control channels to be based on plaintext protocols such as IRC, nowadays, botnets leverage obfuscation and encryption of their C&C messages. This renders methods which use characteristic recurring payload bytes ineffective. In addition, we observe a trend towards distributed C&C architectures and nomadic behavior of C&C servers in botnets with a centralized C&C architecture, rendering blacklists infeasible. Therefore, we identify and recognize botnet C&C channels by help of traffic analysis. To a large degree, our clustering and classification leverage the sequence of message lengths per flow. As a result, our implementation, called CoCoSpot, proves to reliably detect active C&C communication of a variety of botnet families, even in face of fully encrypted C&C messages. Furthermore, we observe that botmasters design their C&C channels in a more stealthy manner so that the identification of C&C channels becomes even more difficult. Indeed, with Feederbot we found a botnet that uses DNS as carrier protocol for its command and control channel. By help of statistical entropy as well as behavioral features, we design and implement a classifier that detects DNSbased C&C, even in mixed network traffic of benign users. Using our classifier, we even detect another botnet family which uses DNS as carrier protocol for its command and control. Finally, we show that a recent trend of botnets consists in rogue visual monetization. Perceptual clustering of Sandnet screenshots enables us to group malware into rogue visual monetization campaigns and study their localization as well as monetization properties.